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ABSTRACT 

The effects of varying the ^seq^uence of frames in a 
self-instructional program on the instructional outcomes of 
achievement, interest in learning, and attitudes toward programed 
instruction were investigated* Thirty-nine 6th grade Spanish classes 
vere randomly assigned to a program vith an orderly progression or a 
scrambled order of frames* The students worked, on the program for 30 
minutes each week without any teacher-directed instruction* For none 
of the three outcomes did the mean levels differ significantly after 
one semester of instruction* in each group a high relationship vas 
found between aptitude and achievement and between initial attitudes 
and interest in learning* Attitudes toward programed instruction were 
not consistently related to any other variables* The conclusion was 
that small variations\in segnence exert little effect on outcomes* 
(KB) ^ ^ 
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THE EFEECTS OF VARI/ITIONS IN.A SEtiF- INSTRUCTIONAL PROGRAM 
ON INSTRUCTIONAL OUTCOMES 

^ Jibs tract 

The effects of varying the sequence of frames in a self" instructional 
program on the instructional outcomes of achievement, interest in learning, 
and attitudes toward programed instruction were investigated, Thirty*nine 
sixth-grade Spanish classes vere randoinly assigned to a program with an 
orderly progression or a scrambled order of frames. The students worked 
on the progi^'am for JO minutes each week without any teacher-directed 
instruction. For none of the three outcomes ^id the mean levels differ 
significantly after one semester of Instruction, In each group a high 
relationship wa^ found between aptitude and achievement and between initial 
attitudes and interest in learning. Attitudes toward programed instruction 
were not consistently related to any other variables. The conclusion was 
that iiiiall variations in sequence exert little effect on outcomes- 



. THE EFFECTS OF VARIATIONS IN A SEUF-INSTRUCTIONAL PROGRAM OK 

INSTBUCTIONAL OUTCOMES^ 

^ One of the current controversies in prograoied instruction is the 
identification of variables that account for learning. Cook and Meohner 
(1962), for example, cite the fdlloving "elements combined by programed 
instruction to produce optimal learning"; 

(a) Active response by the learner; 

(b) Small steps in which ^areful control of stimuli groduces_ gradual 
increments in mastery of the subject; 

(c) Immediate feedback for each response; 

(d) Self -pacing; 

(e) Low error^ate, which is a consequence of the preceding four 
principles. 

They go on to write that- "error- free learning is not only simpler, but 
its effects improve morale, motivation, and retention" (p, 5)* These 
authors imply that the absence of the five principles they cite will result 
in lower levels of both cognitive end noncognitive outcomes- 

The AERA-APA-EAVI Joint Committee on Programed Instruction and Teaching 
Machines {1963), on the other hand, has stated: "At present, the scientific 
evidence is not considered sufficient to • . • justify recommendation that 
adherence to specific i;ules of program construction be used as a basis for 
program evaluation. External evidence is recommended as the main basis for 
the evaluation of program effectiveness." 

Both positions cited here accept the behavior of the learners as the 
criterion by which to evaluate a program, but they differ on Whether the . 
5ri)gram^s internal characteristics -can £>re'dict th±e behavior, 
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The available empiriced evidence tends to support the position of the Joint 
Committee, Studies which have measured the outcome of achievement (e,g, j 
Ashbaughj l$6k'^ Coulson & Silbermanj I960; Evansj Glaser^ & Hommej I960; 
Goldbeck & Campbell^ 1962; Jacobs & Kulkamij 1963; Levin & Baker^ 1963; 
Moore & Smithy 196li-; Eoe^ Case^ & Roe^ 1962 )j have found only minor or non- 
significant effects 'by varying the type of response (overt vs, covert) and 
sequence of the frames (orderly vs, scrambled). 

The present study included the noncognitive outcomes of interest in learning 
and attitudes toward programed instruction^ as well as the cognitive outcome 
of achievement in assessing the effects of frame sequence in a program. The 
study also employed input characteristics in a dual role: first; as predictors 
that may account for jthg^^fferences among the classes in a way that is 
meaningful to educators^ and second; as covariance control variables that 
may increase the statistical precision of the experimental design, 

. Method 

Subjects , The subjects were 82l^ students in 39 sixth-grade classes from 
the Denver Public Schools^ Denver^ Colorado, 

Programs, A linear self- Instructional program of 2016 frames^ with a 
low error rate (5,7?^) and a logiced sequence^ was developed to teach Spanish 
reading and writing to sixth-jgrade classes* . Xhe programing principles 
formulated by Skinner (i960) wei^e carefully followed; first the behavioral 
objectives were specified in advance^ then the frames were written^ tried 
out J and revised until the learners performed satisfactorily. Because of the 
low error rate and the orderly progression of the frames^ this version is 



called the sineLll-step version, The program jwas tried out in several schools 
prior to the present evaluation (BarcuSj Haymanj & Johnsonj 1961). 

The sequence, of frames of the small-step version was altered to produce a 
scrambled ves^sion. The alterations were not random; instead the frames within 
a unit were changed on an intuitive basis to break up the repetitiveness . 
within each unit* The scrambled version was tlius intended to be effective in 
producing learning and in stimulating interest. Its error rate of 7»15t was 
significantly higher (,01 level) than that for the small-step version 
{Hayman & Johnson^ 1963)' ^ 

Procedure , The program^ versions were randomly assigned to classes^ vith 
the restriction that the scrambled version be assigned to 22 classes in order 
to use up the existing supply of programs. 

The classes worked on the programs for half hour each Wednesday 

/ 

for one semester* No homework assignments or other instruction in Spanish 
reading and writing were given. The teachers^ who were told not to answer any 
questions about Spanish^ served only to maintain classroom order and to 
help wit'h the mechanics of the program. Most teachers^ in factj did not know 
any Spanish. 

The students also learned Spanish listening and speaking skills on 
Tuesdays and Thursdays^ via televised and teacher-directed instruction. They 
had previously studied Spanish listening and speaking in the fifth grade, 
but had no prior instruction in Spanish reading and writing. 

Input measures . The Kuhlman -Anderson Intelligence Test was used as the 
measure of academic aprtitude; these scores were available from the school 
records. Initial attitudes toward Spanish (Preattitudes) were measured by 
the four-item inventory reproduced in Appendix A. 
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Outcome measures * Three kinds of instructioned outcomes vere used to 
evaluate the programs: (l) Achievementj (2) Attitudes toward Programed 
Instruction^ and (3) Interest in Spanish, The Sixth Grade Spanish^ Reading 
and Writing Test (Follett Publishing Co,j 196k) was used to ineasure achievement 
at the end of the first semester. The testj which was used with classes 
taught by a variety of other methods as well as by the program alone^ had 
a split-half reliability of ^^k (Hayman & Johnson^ 19^3) • " 

Attitudes toward Programed Instruction were measured by asking the 
students whether they preferred to learn various courses by a program edonej 
by a teacher alone^ or by a combination of teacher plus program (items 2j 
3j and k in Appendix B), Tne scoring of each item was dichotomized; a plus 
one was assigned to the alternative of learning by a program alone^ and a 
Zero was assigned to the other two alternatives* A high score reflected a 
desire to learn the courses by a program alonej a low score reflected a 
desire for teacher-directed instruction^ either alone or in combination with 
a program. The internal consistency'reliabiiity of the attitude score was 
*6o. 

Interest in Spanish was measured by asking the student hov frequently 
and for how many years he would like to study a foreign language^ how much 
he enjoyed Spanish^ and how often he read and conversed in Spanish on his 
ovn outside of school assignments (items 5-17 in Appendix b)* The internal 
consistency reliability of the interest score was ,86, 

Results 

Analyses of covariance were used to compare the effects of the program 
versions on the outcomes of Achievement (with Academic Aptitude as the control 
variable) and interest in Spanish (with Preattitudes as the control 
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variable). Because no input measure was consistently correlated with the 
Attitude toward Programed Instruction scorej the single classification analysis 
of variance was used with this outcome* The clasSj and not the individual 
studentj was the sampling unitj and therefore the class mean served as the 
rniit of observation in the analyses, 

/The results of the analyses of covariance and variance are showix^in 
Table 1. The two versions of the program did not differ significantly in their 
effects on any of the three outcomes* During the one semester of instruction^ 
the sequence of the frames did not ^exert a substantial effect on class 
learning^ interests^ or attitudes; 

liisert Table 1 about here \ 

\ 

\ 

The input characteristics of the class were substantially related to the 

outcomes of Achievement and Interest in Spanish, As shown in Table 2, the 

correlation of Aptitude with Achievement w^s higji in both groups^ ,70 ^for the 

classes taught by the small-step version and ,82 for the classes taught by 

the scrambled version* The correlation of Preattitudes with Interest in. 

Spanish was almost as highj ,50 and *71j respectively. The Attitudes toward 

Programed Instruction score was not correlated with Preattitudes for either 

version; it correlated positively with both Aptitude and Achievement for the 

small- step version^ but negatively with them for the scrambled version* 

These correlations may be a result of chance rather than of systematic effects 

2 

of the program version* % 

Insert Table 2 about here 

/ 

/ 



Discussion 

Contrary to what may be expected from the principles of program development 
as stated by Cook and Mechner (1962)^ the variations in the program had little 
effect on the outcomesj both cognitive and noncognitive. The results of this 
study are consistent with much previous research and with the reccmimendations 
of the AERA-APA-DAVI Joint Committee on Programed Instruction and Teaching 

Machines (1963)* 

\ 

Both versions of the program tended to build upon the initial abilities 

and attitudes of the classes* Classes with high aptitude usually learned the 

mostj and classes with the most favorable initial attitudes tended to have the 

highest level of interests at the end of the semester. Differences in outcomes 

were predictable by differences in input^.but not by differences in the kind of 

program used for instruction* 

In a larger study the small -step version used in combination with a trained 

I 

Spanish teacher produced significantly more achievement thail the program alone 
(Maier & Jacobs^ 1961^), This study also found that interest in learning was 
enhanced (a) by using the prog^^aiQ as homework instead of classworkj and (b) by 
using the program with teacher niirected instruction plus making available a 
special corner of the ^ classroom containing electronic aidsj reading materials^ 
and cultural artifacts (Hayman & Johnson^ l96li-). It appears that the level of 
outcomes^ when not affected by differences in the program itself^ may be 
affected by how the prograiu is used* 

One conclusion suggested by this and other research is that the effective- 
ness of a program may be relatively insensitive to changes within the program* 
To borrow a term from statistics j self -instructional programs may be colled 
robust* In many cases the theoretical assumptions underlying the development 
of programs may not prove too Important* 



References 

\ 

AERA'APA*MVl*Joint Committee on Programed Instruction and Teaching Machines^ 
1962j Interim Report of the Joint Committee on Programed Instruction and 
Teaching Machines. Audiovisual Instruction, 1963, 8, 8k^&9. 

Ashbaughj W. H. Effect on achievement of vritten responses to programed 
learning mterial for students of differing academic ability. 
Psychological Reports, 196k , Ihj 780*782. 

Barcus, D. L., Haymanj J* L., Jr., & Johnson, J. T.jJr^ Development of 

i 

programed learning materials for use with televised-Spanieh instruction. 
Report No. Denver-Stanford Project on the Context of Instructional 
Ttele vision. Denver: Denver Public Schools, I96I. 
Cook, D,, & Mechner, F. Fundamentals of programed instruction. In L. D. Eigen 
and/S. Margulies (Eds.), Applied programed instruction . New York: 
Wiley, 1S62. Pp. 

Coulson, J. E., & Silberman, H. F. Effects of three variables in a teaching 
machine. Journal of Educational Psychology » 196O, 51, 135'l'i-3. 

Evans, J. L*, Glaser, R*, & Homme, L. E. A preliminary investigation of 

variation in the properties of verbal learning ©eminences of the**teaching 
machine** type. In A.- A* Lumsdaine and R. laser (Eds.), Teaching 
machines and _B>^ogramned learning; A source book . Washington, D. C. 
( National Education Association, I960. Pp. 1^^*6-1^51 

Follett Publishing Co. Sixth Grade Spanish, Reading and Writing Test, First 
Semester, Parts 1 and 2. Chicago: Author, 1961*. 

Goldbeck, R. A., & Campbell, V. N. The effect of response mode and response 
difficulty on progx^amed Ijeaming. Journal of Educational Psychology , 
1962, 53, 110-118. 



Haymarij J. L.j Jr.j Johnson^ J. T.j Jr. Reading and writing results in the 
second year of research--196l'62 . Report No. 7j Denver^Stanford Project 
on the Context of Instructional Television. Denver: Denver Public 
SchoolSj 1963* 

J. L.J Jr. J Johnsonj J. T./ Jr. Third y^ar results in the 
Denver-Stanford project. Report No. lOj Denver-Stanford Project on 
the Context of Instructional Television. Denver: Denver Public Schools^. 
196U. 

JacobSj p. I.J & Kulkamij S. A test of some assunrptions underlying programaed 

instruction. Research Bulletin 63-28 • Princeton^ N. J.' Educational 

I * 

Testing Service^ 1963* 
Levinj p. R*j & Bakery G. R. Item scrambling in a self -instructional program. 

Journal of Educational Psycholbgy , 19^3 j 138-llt3. , ^ 
Maierj M. H.j & Jacobs^ P. I. Programed learning: Some recommendations and 

results . The Bulletin of the National Association of Secondaiy-^School 

Pi^ncipals, 1*8^ 2lt2-2^5. 

Moorej J. W.j Smithy W. J. Role of knowledge and knowledge of results in 

programed Instnaction. Psychological Reports, 1961;^ 11^ lt07*H23. 
Roej K. W.J Casej H. W.j & Roe^ A. Scrambled verus ordered sequence in auto' 

instructional jflrograme. ' Journal of Educational Psychology^ 19^2^ ^ 101- 

lOlfr. 

Skinnerj B. F. Teaching machines. In A. A. Inmsdaine and R. Olaser (Ed5.)j 
Teaching machines and progranped learning: A source book * Waahlngtonj 
D. C: National Education Association^ 196O. fti. 137-158. ~ - 




11 

\ 



-9- 

Footnotes 

\?his reBear,ch was supported by a grant from the Carnegie Corporation of 
New York to Educational Testing Service, The author? wish to thank Drs, 

Herbert Gerjuoy and Felix Kbpstein for their helpful comments In reviewing 

/ ' - 

an earlier version of the paper, 
2 

As part of a l*£rger study^ the small-step version of the program was also 
used in combination with a trained Spanish teacher For I5 classes taught by 
the combination of teacher plus program^ the Attitude toward Programed 
Instruction score correlated with Aptitude anS -;»53 w^th Achievement, 
Since there is no obvious substantive explanation of why the correlations 
should vary so, the bes't explanation may be that the, cofrelafcionr arose by 
chance. 
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Table 1 
Analysis of OutcoiDes 



Outcome Source of Variation DF « F 

square 

A, Achievement Erograiu Versions 1 25*6 l^k HS 

(Predictor : Aptitude) 

Error , 36 18,3 



Interest In Spanish Erograia Versions 1 ,005 *00k NS 

(Predictor: Ereattltudes) 

Error 36 1*3 

i 

\ 

C, Attitudes toward Erograraed Program Versions 1 ,025 *^ ' 

Ins:t ruction 

(Predictor: none) Error 37 '063 



ft 

Adjusted Mean Squares shown for the outcomes of Achievement and 
Interest In Spanish , 



\ Table 2 

Results for Two Versions of a 
Self -Instructional. Program 



A. Small-step Version (N = 17 Classes) 







Input ~" 
1 2 


3 


Outcome 


■ / 

5 


Mean 


S.D. 


1. 


Aptitude 




-01 


70 


oi^ 


75 


106.5 


6.6 


2. 


Preattitude 


-01 




or 


50 


-08 


1^.8 


.51^ 


3. 


Achievement ' 


70 


or 




-06 


71^ 


20.5 


1^.7 




Interest in Spanish 


oi^ 


50 


-06 




-11 


10.5 


■ 1.1 


5.. 


Programeii Instruction 


T5 


-08 


71^ 


-11 




.1^7 


.21 



■ Attitudes 



B. Scrambled Version (N - 22 Classes) 



t 




Input 
1 2 


3 


Outcome 




Mean , 


-1' ' 
S.D. 


1. 


1 ^ 

1 ^ 

Aptitude \ 




. -02 


82 


-19 


-32 


103.5 


8.3 


2. 


Preattitude 


-02 




21 


71 


-03 


1^.7 


.66 


3. 


V 

Achievement 


82 


21 




16 


-27 


20.2 ' 


7.8 




Interest in Spanish 


-19 


. 71 


16 




-01 ~ 


10.3 


1.6 


5. 


programed Instruction 


-32 


-03 


-27 


-01 , 




•52 


.27 



Attitudes 



\ L_ 

Note: - DeciLmal points have been omitted from the correlations. 
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D More than just this year 
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Appendix A 

\ ' / Sixth Grade Enrollment Forta - Page 3 ' 

The following four questions deal with your opinions a'bout talcing Spanish, 
Please check one answer for each question — whichever best describes hov you 
feel. Remember J your answer will not have any bearing on your marks or your 
standing in the classj so please answer honestly, 
16, Hov long would you lil^ to study Spanish? 

D I would like to drop Spanish right now 

D Just this year J but no more _ 
More thar 

17* Do you think Spanish should be taught in the sixth grade? 

□ No - ^ 

□ Yes ■ 

D I'm not sure \^ 

18, Hov^inuch did you enjoy Spanish last year? ; 
D About the same as iny other subjects 
D More than iny other subjects 
D Less than my other subjects 
D I did not study Spanish last year 

19, Did studying Spanish help you in English? 

□ No 

□ Yes -7 
D I 'in not sure , 
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Your Name Teacher's Name ' 

School 

STUDENT QUESTIONNAIEE 

/ 

^Last year in the fifth grade you learned how to speak Spanish* Now as 
sixth graders you are learning how to read and vrite Spanish* Some of you are 
leading to read and write Spanish from your teacher j others are learning from 
Automated Spanish j a prograimned textbook^ and still others from your teacher 
and prograjmned textbooks* 

Here are some questions about Spanish and about the way you ai*e learning it* 

Read each question and the possible answers* 
Decide on your answer* * 

Put a check mark like this next to the answer you choose* 
Make ysure you answer each question* . _ ^ 

u now learning to re^ and write Spanish? 
a* From n^y teacher 
b* From prograimned textbooks 
c * From my teacher and programmed textbooks * 




2* If you had your choice^ how would you want to learn Spanish reading and 
writing? 

a* From my teacher 

b* From programmed textbooks 

c * From my teacher and programmed textbooks * 



If you had your choice^ how would you wa^t to learn arithnietic? 
\ a* From my teacher 



b* From programmed textboolfs 

c . From i^y teacher and pr^^aramed textbooks 



k* If you had your chcdce^ how would you "jrant to study English ? 

a* From my teacher / 

b * From prograiamed textb/ooks 

^ c * From my teacher and programmed textbooks 



i 



16 



ERIC 



-B2- 



5. If you had your choicej what foreign language would you like to learn in 
tne sixth grade? 

a. French 

b. German 

c. Latin 

d. Spanish 

e. Russian 

f . No foreign language 



6. If you had your choice^ what foreign language would you like to learn in 
high school? 

a. French 

t . Gertnan 

c^. Latin 

d. Spanish 

, e. Russian 

f . No foreign language 



7v How much did you enjoy learning to speak Spanish last year? 

a. More than my other subjects 

b. About the same as my other subjects 

J c. Less, than my other subjects 

8. How much do you enjoy learning to read and write Spanish this year? 

a. More than my other subjects 

b. About the same as my other st&jects 

c. Less than my^ other subjects 

9, How often do you -read Spanish newspaporsj stories^ and so forth on your own? 
Do not include your rea<iing for class assignments. 

a. Rarely or never 

b. Once in a while 

c. Often 



10, How often do you translate Spanish on your own? 



a. Rarel^^^or never 

b . Once in a while 

c. Often 
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11. Hovf often do you thinlc in Spanish when you are not working on your class 

assignments? 

a. Rarely or never 

. b . Once in a while 

c . Often 

12. How often do you sp eak to your friends in Spanish? 

a. Rarely or never 

_^ b. Once in a while 

c. Often 



13. How often do you speak to your parents in Spanish? 

a. Rarely or liever 

b. Once in a while 

c. Often 

ik. How often would you like to take Spanish this year? 

a. Every day ' 

b . k times a week 

c . 3 times a week 

d. Twice a week 

e. Once a week 

^ Not at all 

15. How long would you like to study Spanish? 

a. I would like to drop Spanish right now ^ 

b. Just this year^ but nb more 

c More than Just this year 

16 . Do you think studying Spanish helps you with English? 

, a. No ^ 

b. Yes 

c. I'm not sure 

17. Do you think studying Spanish helps you figure out the meaning of new 

English words? 

a. Mo , * 

b. Yes ■ 

c. I'm not sure 
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